Skip to content

feat: kafka-connect-monitor service#3552

Merged
epipav merged 3 commits intomainfrom
feat/kafka-connect-monitoring
Oct 31, 2025
Merged

feat: kafka-connect-monitor service#3552
epipav merged 3 commits intomainfrom
feat/kafka-connect-monitoring

Conversation

@epipav
Copy link
Copy Markdown
Collaborator

@epipav epipav commented Oct 29, 2025

Summary

  • Adds a new Kafka Connect monitoring service that exposes Prometheus metrics for connector and task health
  • Provides real-time health monitoring capabilities for Kafka Connect via /connector-health endpoint
  • Enables alerting and observability for Kafka Connect infrastructure
  • It is running as a sidecar to kafka-connect service

Implementation Details

Created a new standalone service kafka-connect-monitor that:

  • Polls Kafka Connect REST API at localhost:8083 to fetch connector statuses
  • Exposes Prometheus metrics using one-hot encoding for connector and task states (RUNNING, FAILED, PAUSED, UNASSIGNED)
  • Runs on port 8085 with health check endpoint at /health
  • Built with Express and prom-client for metrics collection

@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

cursor[bot]

This comment was marked as outdated.

@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

@epipav epipav requested a review from themarolt October 29, 2025 13:41
@github-actions
Copy link
Copy Markdown
Contributor

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

  • feat: add user authentication (CM-123)
  • feat: add user authentication (IN-123)

Projects:

  • CM: Community Data Platform
  • IN: Insights

Please add a Jira issue key to your PR title.

request.log.error(err, 'Error occurred!')
res.status(500).send('Internal Server Error')
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Middleware Order Causes Error Handling Failure

The errorMiddleware tries to use req.log for error reporting. However, loggingMiddleware is placed after express.json(). If JSON parsing fails, req.log won't be initialized, leading to a TypeError when the error handler attempts to access req.log.error().

Additional Locations (1)

Fix in Cursor Fix in Web

@epipav epipav merged commit 3804abe into main Oct 31, 2025
14 checks passed
@epipav epipav deleted the feat/kafka-connect-monitoring branch October 31, 2025 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants